Motion image coding apparatus, and control method and program of the apparatus

ABSTRACT

A motion detector detects the motion of an imaging device. On the basis of the detected motion information, a selector selects, from a plurality of frame memories for storing a plurality of frame data, a frame memory for storing reference frame data to be referred to when frame data to be coded is coded. A motion estimator estimates a motion vector on the basis of the reference frame data stored in the selected frame memory and the frame data to be coded. The frame data to be coded is coded by using the estimated motion vector, and the coded data is output.

TECHNICAL FIELD

The present invention relates to a motion image coding apparatus whichcodes a motion image by executing motion compensation for frame data tobe coded by referring to a plurality of frame data in the motion image,and a control method and program of the apparatus.

BACKGROUND ART

Recently, the H.264 coding method is attracting attention as a newmotion image coding method. This coding method is developed by thecooperation of ITU-T and ISO. This new standard was standardized in thesummer of 2003.

The characteristic features of this new coding method are that 4×4integer transformation is used, and a plurality of intra-predictions areprepared, unlike in the conventional MPEG-1, MPEG-2, and MPEG-4 codingmethods. In addition, an intra-loop filter is used, and motioncompensation is performed by seven types of sub-blocks. Also, the pixelaccuracy of the motion compensation is the same as the MPEG-4 codingmethod, i.e., the motion compensation can be performed by 1/4 pixelaccuracy. Furthermore, universal variable-length coding or contextadaptive variable-length coding is used as entropy coding.

The more important characteristic feature is as follows. That is, theMPEG-1, MPEG-2, and MPEG-4 coding methods perform motion compensation byusing two reference images (frames) before and after a frame to becoded. However, this new coding method can use a larger number ofreference images. A num_ref_frames code contained in the header of a bitstream can take a maximum of 16 values.

More specifically, in motion compensation, 16 frames before and after aframe to be coded can be referred to as reference images. A macroblockto be coded is processed as follows. As described above, a predictionerror is calculated by ¼ pixel accuracy for seven types of sub-blockswith respect to an image having a maximum of 16 frames, and a macroblockby which this prediction error is a minimum is selected. This largelyincreases the coding efficiency.

The arrangement of the conventional motion image coding apparatus usingthe H.264 coding method will be explained with reference to FIG. 13.This arrangement is also explained in reference 1 (“Overview of theH.264/AVC Video Coding Standard” (IEEE TRANSACTIONS ON CIRCUITS ANDSYSTEMS FOR VIDEO TECHNOLOGY, JULY 2003)) or reference 2 (“H.264 NowStarts, Outrivaling Slow-Going MPEG-4” (Nikkei Electronics 2003.7.7, pp.65 -74)).

FIG. 13 is a block diagram showing the arrangement of the conventionalmotion image coding apparatus.

Image data is input macroblock by macroblock to this motion image codingapparatus. A selector 1000 switches whether to perform intra-coding. Ifintra-coding is to be performed, the image data is input to anintra-predictor 1001. The intra-predictor 1001 performs prediction innine modes, and calculates a prediction error.

If coding to be performed is not intra-coding, the image data is inputto a differential unit 1002 where the difference from a predicted imageis calculated as a prediction error.

A transformer/quantizer 1003 transforms the calculated prediction errorinto an integer of 4×4 pixel blocks, and quantizes the obtainedcoefficient. This quantized efficient as the result of quantizationundergoes variable-length coding performed by an entropy coder 1004, andis output to an output unit 1014. At the same time, the quantizationresult is input to a dequantizer/invert transformer 1005 to restore theprediction error, and this prediction error is added to the predictedimage by an adder 1006. The result is suitably stored as a decoded imagein frame memories 1007 to 1010.

A motion estimator 1011 compares the decoded image stored in the framememories 1007 to 1010 with the input image, and calculates a motionvector by ¼ pixel accuracy for each sub-block. These motion vectors andthe selected frame numbers are input to a motion compensator 1012, andreference images are loaded from the corresponding frame memories. Areference image having a minimum prediction error is selected and outputas a predicted image to the differential unit 1002.

The motion vectors and selected frame numbers are also input to a motioncoder 1013 and coded, and the coded data is output to the output unit1014. The output unit 1014 shapes this coded data in accordance with aformat, and outputs the shaped data.

Unfortunately, a coding method which refers to a plurality of framessuch as the H.264 coding method described above poses the problem thatmotion vectors are searched for in order to execute motion compensation,and the calculation amount becomes enormous as the number of referenceimages increases.

In addition, especially when the H.264 coding method is used in an imagesensing apparatus such as a video camera, the entire image largelychanges whenever the video camera is panned or tilted. Therefore,although the frequency of reference of data of temporally separatedframe images decreases, this data must be held.

DISCLOSURE OF INVENTION

The present invention has been made in consideration of the abovesituation, and has as its object to provide a motion image codingapparatus capable of efficiently using a memory used for motion imagecoding, and capable of performing efficient motion vector search, and acontrol method and program of the apparatus.

According to the present invention, the foregoing object is attained byproviding a motion image coding apparatus which codes a motion image byexecuting motion compensation for frame data to be coded by referring toa plurality of frame data in the motion image, comprising: detectingmeans for detecting a motion of an imaging device; a plurality ofstorage means for storing the plurality of frame data; selecting meansfor selecting, from the plurality of storage means, on the basis ofmotion information detected by the detecting means, storage means forstoring reference frame data to be referred to when the frame data to becoded is coded; estimating means for estimating a motion vector on thebasis of the reference frame data stored in the storage means selectedby the selecting means and the frame data to be coded; coding means forcoding the frame data to be coded by using the motion vector estimatedby the estimating means; and output means for outputting the coded datawhich is coded by the coding means.

In a preferred embodiment, the detecting means detects the motion of theimaging device on the basis of a motion image sensed by the imagingdevice.

In a preferred embodiment, wherein the selecting means comprises controlmeans for controlling write/read and power supply to the plurality ofstorage means on the basis of the motion information detected by thedetecting means.

In a preferred embodiment, further comprises setting means for settingan image sensing mode of the imaging device, wherein the selecting meanscomprises control means for controlling write/read and power supply tothe plurality of storage means on the basis of the image sensing modeset by the setting means.

In a preferred embodiment, the control means stops power supply tostorage means not selected by the selecting means.

In a preferred embodiment, the apparatus further comprises search rangecontrol means for controlling a motion vector search range of theestimating means on the basis of the motion image detected by thedetecting means.

According to the present invention, the foregoing object is attained byproviding a motion image coding apparatus which codes a motion image byexecuting motion compensation for frame data to be coded by referring toa plurality of frame data in the motion image, comprising: setting meansfor setting an image sensing mode of an imaging device; a plurality ofstorage means for storing the plurality of frame data; selecting meansfor selecting, from the plurality of storage means, on the basis of theimage sensing mode set by the setting means, storage means for storingreference frame data to be referred to when the frame data to be codedis coded; estimating means for estimating a motion vector on the basisof the reference frame data stored in the storage means selected by theselecting means and the frame data to be coded; coding means for codingthe frame data to be coded by using the motion vector estimated by theestimating means; and output means for outputting the coded data whichis coded by the coding means.

In a preferred embodiment, the selecting means comprises control meansfor controlling write/read and power supply to the plurality of storagemeans.

In a preferred embodiment, the control means stops power supply tostorage means not selected by the selecting means.

In a preferred embodiment, the detecting means detects a motion of theimaging device on the basis of a motion image sensed by the imagingdevice.

According to the present invention, the foregoing object is attained byproviding a motion image coding apparatus which codes a motion image byexecuting motion compensation for frame data to be coded by referring toa plurality of frame data in the motion image, comprising: input meansfor inputting control information which controls an imaging device;storage means for storing a motion image sensed by the imaging device;setting means for setting the number of reference frame data to bereferred to when the frame data to be coded is coded, on the basis ofmotion information of the imaging device, which is acquired on the basisof the control information input by the input means; acquiring means foracquiring reference frame data corresponding to the number of referenceframe data set by the setting means; estimating means for estimating amotion vector on the basis of the reference frame data acquired by theacquiring means and the frame data to be coded; coding means for codingthe frame data to be coded by using the motion vector estimated by theestimating means; and output means for outputting the coded data whichis coded by the coding means.

In a preferred embodiment, the apparatus further comprises search rangecontrol means for controlling a motion vector search range of theestimating means on the basis of the motion information.

According to the present invention, the foregoing object is attained byproviding a control method of a motion image coding apparatus whichcomprises a plurality of storage units for storing a plurality of framedata in a motion image, and codes the motion image by executing motioncompensation for frame data to be coded by referring to frame datastored in the plurality of storage units, comprising: a detection stepof detecting a motion of an imaging device; a selection step ofselecting, from the plurality of storage units, on the basis of motioninformation detected in the detection step, a storage unit for storingreference frame data to be referred to when the frame data to be codedis coded; an estimation step of estimating a motion vector on the basisof the reference frame data stored in the storage unit selected in theselection step and the frame data to be coded; a coding step of codingthe frame data to be coded by using the motion vector estimated in theestimation step; and an output step of outputting the coded data whichis coded in the coding step.

According to the present invention, the foregoing object is attained byproviding a control method of a motion image coding apparatus whichcomprises a plurality of storage units for storing a plurality of framedata in a motion image, and codes a motion image by executing motioncompensation for frame data to be coded by referring to frame datastored in the plurality of storage units, comprising: a setting step ofsetting an image sensing mode of an imaging device; a selection step ofselecting, from the plurality of storage units, on,the basis of theimage sensing mode set in the setting step, a storage unit for storingreference frame data to be referred to when the frame data to be codedis coded; an estimation step of estimating a motion vector on the basisof the reference frame data stored in the storage unit selected in theselection step and the frame data to be coded; a coding step of codingthe frame data to be coded by using the motion vector estimated in theestimation step; and an output step of outputting the coded data whichis coded in the coding step.

According to the present invention, the foregoing object is attained byproviding a control method of a motion image coding apparatus whichcomprises a storage unit for storing a motion image, and codes themotion image by executing motion compensation for frame data to be codedby referring to frame data stored in the storage unit, comprising: aninput step of inputting control information which controls an imagingdevice; a setting step of setting the number of reference frame data tobe referred to when the frame data to be coded is coded, on the basis ofmotion information of the imaging device, which is acquired on the basisof the control information input in the input step; an acquisition stepof acquiring reference frame data corresponding to the number ofreference frame data set in the setting step; an estimation step ofestimating a motion vector on the basis of the reference frame dataacquired in the acquisition step and the frame data to be coded, acoding step of coding the frame data to be coded by using the motionvector estimated in the estimation step; and an output step ofoutputting the coded data which is coded in the coding step.

According to the present invention, the foregoing object is attained byproviding a program for implementing control of a motion image codingapparatus which comprises a plurality of storage units for storing aplurality of frame data in a motion image, and codes the motion image byexecuting motion compensation for frame data to be coded by referring toframe data stored in the plurality of storage units, comprising programcodes of: a detection step of detecting a motion of an imaging device; aselection step of selecting, from the plurality of storage units, on thebasis of motion information detected in the detection step, a storageunit for storing reference frame data to be referred to when the framedata to be coded is coded; an estimation step of estimating a motionvector on the basis of the reference frame data stored in the storageunit selected in the selection step and the frame data to be coded; acoding step of coding the frame data to be coded by using the motionvector estimated in the estimation step; and an output step ofoutputting the coded data which is coded in the coding step.

According to the present invention, the foregoing object is attained byproviding a program for implementing control of a motion image codingapparatus which comprises a plurality of storage units for storing aplurality of frame data in a motion image, and codes a motion image byexecuting motion compensation for frame data to be coded by referring toframe data stored in the plurality of storage units, comprising programcodes of: a program code of a setting step of setting an image sensingmode of an imaging device; a program code of a selection step ofselecting, from the plurality of storage units, on the basis of theimage sensing mode set in the setting step, a storage unit for storingreference frame data to be referred to when the frame data to be codedis coded; an estimation step of estimating a motion vector on the basisof the reference frame data stored in the storage unit selected in theselection step and the frame data to be coded; a coding step of codingthe frame data to be coded by using the motion vector estimated in theestimation step; and an output step of outputting the coded data whichis coded in the coding step.

According to the present invention, the foregoing object is attained byproviding a program for implementing control of a motion image codingapparatus which comprises a storage unit for storing a motion image, andcodes the motion image by executing motion compensation for frame datato be coded by referring to frame data stored in the storage unit,comprising program codes of: an input step of inputting controlinformation which controls an imaging device; a setting step of settingthe number of reference frame data to be referred to when the frame datato be coded is coded, on the basis of motion information of the imagingdevice, which is acquired on the basis of the control information inputin the input step; an acquisition step of acquiring reference frame datacorresponding to the number of reference frame data set in the settingstep; an estimation step of estimating a motion vector on the basis ofthe reference frame data acquired in the acquisition step and the framedata to be coded; a coding step of coding the frame data to be coded byusing the motion vector estimated in the estimation step; and an outputstep of outputting the coded data which is coded in the coding step.

Other features and advantages of the present invention will be apparentfrom the following description taken in conjunction with theaccompanying drawings, in which like reference characters designate thesame or similar parts throughout the figures thereof.

BRIEF DESCRIPTION OF DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention and,together with the description, serve to explain the principles of theinvention.

FIG. 1 is a block diagram showing the arrangement of a motion imagecoding apparatus according to the first embodiment of the presentinvention;

FIG. 2 is a flowchart showing a motion image coding process performed bythe motion image coding apparatus according to the first embodiment ofthe present invention;

FIG. 3 is a block diagram showing the arrangement of a motion imagecoding apparatus according to the second embodiment of the presentinvention;

FIG. 4 is a flowchart showing a motion image coding process performed bythe motion image coding apparatus according to the second embodiment ofthe present invention;

FIG. 5 is a flowchart showing details of frame memory control Maccording to the second embodiment of the present invention;

FIG. 6 is a flowchart showing details of frame memory control Naccording to the second embodiment of the present invention;

FIG. 7 is a block diagram showing the arrangement of a motion imagecoding apparatus according to the third embodiment of the presentinvention;

FIG. 8 is a flowchart showing a motion image coding process performed bythe motion image coding apparatus according to the third embodiment ofthe present invention;

FIG. 9 is a block diagram showing the arrangement of a motion imagecoding apparatus according to the fourth embodiment of the presentinvention;

FIG. 10 is a view showing the data configuration of a memory accordingto the fourth embodiment of the present invention;

FIG. 11 is a flowchart showing processing executed by the motion imagecoding apparatus according to the fourth embodiment of the presentinvention;

FIG. 12 is a flowchart showing details of processing in step S305according to the fourth embodiment of the present invention; and

FIG. 13 is a block diagram showing the arrangement of the conventionalmotion image coding apparatus.

BEST MODE FOR CARRYING OUT THE INVENTION

Preferred embodiments of the present invention will be described indetail in accordance with the accompanying drawings.

<First Embodiment>

FIG. 1 is a block diagram showing the arrangement of a motion imagecoding apparatus according to the first embodiment of the presentinvention.

The first embodiment will be explained below by taking a camcoder as anexample of the motion image coding apparatus.

In the first embodiment, a coding method used by the motion image codingapparatus will be explained by taking the H.264 coding method as anexample. However, the coding method is not limited to this method. Also,to simplify the explanation, forward prediction by which frames in thepast are referred to will be taken as an example. In addition, themaximum number of reference frames is set to 5 in order to simplify theexplanation, but this number is not limited to 5.

Referring to FIG. 1, an imaging device 1 generates a digital signal ofan image in cooperation with an optical unit made up of lenses and thelike. A frame memory 2 stores the digital signal. A selector 3 selectsan output from the frame memory 2 in accordance with whether the codingmode is intra-frame coding or inter-frame coding.

An intra-predictor 4 executes intra-prediction by the H.264 codingmethod. A differential unit 5 calculates a motion prediction error. Atransformer/quantizer 6 executes integer type orthogonal transformation,and quantizes the obtained coefficient. A dequantizer/invert transformer10 dequantizes the quantized coefficient, and executes integer typeorthogonal inversion.

An entropy coder 7 codes the quantization result from thetransformer/quantizer 6. A recorder 8 records the coded data on arecording medium 9. The recording medium 9 records the coded data. Anadder 11 adds the prediction error obtained by the dequantizer/inverttransformer 10 and a predicted value (predicted image).

Frame memories 12, 13, 14, 15, 16, and 17 store frames of locallydecoded image data. A selector 18 controls inputs and outputs.

A motion estimator 19 extracts an optimum motion vector from thecorresponding frame on the basis of the input image and decoded image. Amotion compensator 20 generates a predicted image from the motion vectorcalculated by the motion estimator 19 and the corresponding frameinformation.

A motion coder 21 codes motion information on the basis of thecalculated motion vector and the corresponding frame information. Amotion detector 22 detects the motion of the motion image codingapparatus by using a gyro, sensor, or the like. A motion determinator 23determines the speed of the motion, which is detected by the motiondetector 22, of the motion image coding apparatus.

The motion image coding apparatus shown in FIG. 1 further comprises aCPU which controls the entire apparatus, a ROM which stores variouscontrol programs for controlling the apparatus, and a RAM whichfunctions as a work area and temporary save area of various data forexecuting various control operations.

The motion image coding operation of the motion image coding apparatusshown in FIG. 1 will be explained below.

Before coding, the entropy coder 7 generates header informationcontaining, e.g., the number of frames which can be referred to, andrecords this header information on the recording medium 9 via therecorder 8.

Image data of frames sensed by the imaging device 1 is stored in theframe memory 2, and input to the selector 3. The selecting operation ofthe selector 3 is so controlled as to execute intra-frame coding atpredetermined intervals, and- execute inter-frame coding in otherperiods.

First, the execution of intra-frame coding for the first frame as astart frame will be explained below.

The input frame data is input macroblock by macroblock to theintra-predictor 4, and the intra-predictor 4 executes intra-predictionfor each block. The prediction result is input to thetransformer/quantizer 6, and the transformer/quantizer 6 executesinteger type orthogonal transformation, and quantities the obtainedcoefficient. This quantized coefficient as the quantization result isinput to the entropy coder 7 and dequantizer/invert transformer 10.

The entropy coder 7 entropy-codes the input quantization result, andrecords the coded data on the recording medium 9 via the recorder 8.

The dequantizer/invert transformer 10 obtains a decoded image from theinput quantization result, and inputs this decoded image to the adder11. Intra-frame coding uses no predicted image. Therefore, the adder 11adds a predicted value 0, and stores the image in an empty frame memoryor in a frame memory storing the oldest image data of reference imagesrecorded in the frame memories. Initially, no image data is stored inthe frame memories 12 to 17, so the image is stored in the frame memory12.

The execution of inter-frame coding for the subsequent second frame willbe described below.

The motion detector 22 detects the motion of the motion image codingapparatus itself. For example, the motion detector 22 calculatesvertical and horizontal motion vectors MVx and MVy, and defines thesquare sum of these vectors as the magnitude of the motion of the motionimage coding apparatus. The motion determinator 23 determines whetherthe magnitude of the motion of the motion image coding apparatus islarger than a predetermined value.

If the magnitude of the motion of the motion image coding apparatus islarger than the predetermined value, the motion determinator 23 controlsthe selector 18 so as to select outputs from the frame memories 12 to 17such that two temporally newest frames from the frame memories 12 to 17are used as reference images. If the magnitude of the motion of themotion image coding apparatus is equal to or smaller than thepredetermined value, the motion determinator 23 controls the selector 18so as to select all outputs from the frame memories 12 to 17.

The motion estimator 19 reads out the contents of the selected framememory, and calculates a motion vector. The motion compensator 20generates a predicted image, and inputs this image to the differentialunit 5. The selector 3 selects the differential unit 5 as an output. Thedifferential unit 5 calculates a prediction error. The calculationresult is inputted to the transformer/quantizer 6. Integer typeorthogonal transformation is executed and quantized the obtainedcoefficient at the transformer/quantizer 6. The quantized coefficient asthe quantization result is input to the entropy coder 7 anddequantizer/invert transformer 10.

The entropy coder 7 entropy-codes the input quantization result, andrecords the coded data on the recording medium 9 via the recorder 8.Also, the motion vector calculated by the motion estimator 19 is codedby the motion coder 21, and the coded data is recorded on the recordingmedium 9 via the recorder 8.

The dequantizer/invert transformer 10 obtains a prediction error fromthe input quantization result, and inputs the prediction error to theadder 11. The adder 11 adds the predicted image (predicted value) fromthe motion compensator 20 to the prediction error, and stores the sum inan empty frame memory or in a frame memory storing the oldest image dataof reference images recorded in the frame memories.

The process flow of the motion image coding process of the motion imagecoding apparatus according to the first embodiment will be describedbelow with reference to FIG. 2.

FIG. 2 is a flowchart showing the motion image coding process of themotion image coding apparatus according to the first embodiment of thepresent invention.

First, in step S1, header information is generated and output. In stepS2, whether coding of all frames is complete is determined. If coding iscomplete (YES in step S2), the process is terminated. If coding is notcomplete (NO in step S2), the flow advances to step S3.

In step S3, frame data to be coded is input. In step S4 , the motion ofthe motion image coding apparatus is detected.

In step S5, the magnitude of the detected motion of the motion imagecoding apparatus is compared with a predetermined value. If themagnitude of the motion is equal to or smaller than the predeterminedvalue (NO in step S5), the flow advances to step S6 to set the number ofreference frames to M (in this case, 5). If the magnitude of the motionis larger than the predetermined value (YES in step S5), the flowadvances to step S7 to set the number of reference frames to N (in thiscase, 2). In this embodiment, M>N always holds.

In step S8, the coding mode of the frame data to be coded is determined.

If the coding mode is intra-frame coding, the flow advances to step S9to execute intra-frame coding for the input frame data. After that, theflow advances to step S11.

If the coding mode is inter-frame coding, the flow advances to step S10to execute inter-frame coding by executing motion compensation whileimages in frame memories equal in number to the reference frames arereferred to. After that, the flow advances to step S11.

In step S11, the decoded image is stored in an empty frame memory or ina frame memory storing the temporally oldest image data of the framememories.

In step S12, the coded data is recorded on the recording medium 9. Then,the flow returns to step S2 to process the next frame.

In the first embodiment as described above, if the motion of the imagingdevice is large (lively), the frequency of reference of temporallyseparated frames extremely decreases, so these frames are not referredto. This largely reduces the arithmetic processing amount pertaining tomotion compensation. As a consequence, the processing speed can beincreased.

In the first embodiment, the maximum number of reference frames is setto 5. However, it is obviously also possible to use any number ofreference frames by using a necessary number of frame memories.

Also, in the first embodiment, the number of reference frames when themotion is large is not limited to the above-mentioned number. Inaddition, the number of frames to be selected may also be variable inaccordance with the magnitude of the motion. For example, the number offrames which can be referred to can be determined inversely proportionalto the magnitude of the motion of the motion image coding apparatus, orsome other standard may also be used.

In the first embodiment, the motion of the imaging device 1 is detectedby the motion detector 22 (sensor). However, the motion of the imagingdevice 1 may also be detected on the basis of input image data. Forexample, it is also possible to detect a rough motion vector bygenerating a reduced image, or detect the motions of several points ofan image and use the detection results as the motion of the imagingdevice 1.

The first embodiment is explained by taking only forward prediction asan example. However, backward prediction or bidirectional prediction mayalso be performed by using a necessary number of frame memories.

It is, of course, also possible to describe, in software, some or all ofthe various constituent elements of the motion image coding apparatusaccording to the first embodiment, and execute processing by anarithmetic unit such as a CPU.

<Second Embodiment>

FIG. 3 is a block diagram showing the arrangement of a motion imagecoding apparatus according to the second embodiment of the presentinvention.

The same reference numerals as in FIG. 1 of the first embodiment denoteparts having the same functions in FIG. 3, and an explanation thereofwill be omitted.

Referring to FIG. 3, a motion estimator 100 refers to an input image andframe data which can be referred to from decoded image data in framememories 12 to 17, and extracts an optimum motion vector from thecorresponding frame data.

A motion compensator 101 generates a predicted image from the motionvector calculated by the motion estimator 100 and the correspondingframe information. A frame memory controller 102 controls the framememories 12 to 17. That is, the frame memory controller 102 controlswrite and read and the supply of power to the frame memories 12 to 17.

The motion image coding operation of the motion image coding apparatusshown in FIG. 3 will be described below.

First, as in the first embodiment, header information is generated andrecorded on a recording medium 9 via a recorder 8.

Then, as in the first embodiment, image data of frames sensed by animaging device 1 are stored in a frame memory 2 and input to a selector3.

When intra-frame coding is to be executed, processing is entirely thesame as in the first embodiment, and the generated coded data isrecorded on the recording media 9 via the recorder 8.

In addition, in accordance with instructions from the frame memorycontroller 102, a decoded image is stored in an empty frame memory or ina frame memory storing the oldest image data of reference imagesrecorded in the frame memories. That is, the frame memory controller 102supplies power to one of unused frame memories to set it in a writablestate, or sets a frame memory having the oldest frame of the framememories in a writable state.

The execution of inter-frame coding will be explained below.

As in the first embodiment, a motion detector 22 detects the motion ofthe motion image coding apparatus itself. For example, the motiondetector 22 calculates vertical and horizontal motion vectors MVx andMVy, and defines the square sum of these vectors as the magnitude of themotion of the motion image coding apparatus. A motion determinator 23determines whether the magnitude of the motion of the motion imagecoding apparatus is larger than a predetermined value.

If the magnitude of the motion of the motion image coding apparatus islarger than the predetermined value, the motion determinator 23 notifiesthe frame memory controller 102 that the motion image coding apparatushas moved largely. When receiving this notification, the frame memorycontroller 102 sets two of the frame memories 12 to 17 in a readablestate, and sets another frame memory in a writable state, in order toselects outputs from the frame memories 12 to 17 so that the twotemporally newest frames are chosen as reference images. In addition,the frame memory controller 102 stops the supply of power to the otherframe memories.

If the magnitude of the motion of the motion image coding apparatus isequal to or smaller than the predetermined value, the motiondeterminator 23 notifies the frame memory controller 102 that the motionimage coding apparatus has not largely moved. When receiving thisnotification, the frame memory controller 102 sets a frame memorystoring the frame data in a readable state. Also, the frame memorycontroller 102 supplies power to one of unused frame memories to set itin a writable state, or sets a frame memory having the oldest frame ofthe frame memories in a writable state.

The motion estimator 100 reads out the contents of the frame memory setin the readable state, and calculates a motion vector. The motioncompensator 101 generates a predicted image, and inputs the image to adifferential unit 5. After that, as in the first embodiment, inter-framecoding is performed, and the coded data is recorded on the recordingmedium 9 via the recorder 8.

A dequantizer/invert transformer 10 obtains a prediction error from theinput quantization result, and inputs this prediction error to an adder11. The adder 11 adds the predicted image from the motion compensator101 to the prediction error, and stores the addition result in the framememory set in the writable state.

The process flow of the motion image coding process of the motion imagecoding apparatus according to the second embodiment will be describedbelow with reference to FIGS. 4 to 6.

FIG. 4 is a flowchart showing the motion image coding process of themotion image coding apparatus according to the second embodiment of thepresent invention.

The same reference numerals as in FIG. 2 of the first embodiment denotesteps having the same functions in FIG. 4, and an explanation thereofwill be omitted.

As in the first embodiment, after processes in steps S1 to S4, if themagnitude of the motion is equal to or smaller than the predeterminedvalue in step S5 (NO in step S5), the flow advances to step S100 toexecute frame memory control M. If the magnitude of the motion is largerthan the predetermined value (YES in step S5), the flow advances to stepS101 to execute frame memory control N. In this embodiment, M>N alwaysholds.

Details of frame memory control M in step S100 will be explained belowwith reference to FIG. 5.

FIG. 5 is a flowchart showing the details of frame memory control Maccording to the second embodiment of the present invention.

In step S102, the number of reference frames is set to M (in this case,5). In step S103, the presence/absence of unused frame memories whichare supplied with no power and kept unused is determined. If there is nounused frame memory (NO in step S103), the flow advances to step S104 toset a frame memory storing the temporally oldest image data (referenceframe) in a writable state.

On the other hand, if unused frame memories exist (YES in step S103),the flow advances to step S105 to select one unused frame memory andmake it usable by supplying power to it. In step S106, this frame memorymade usable in step S105 is set in a writable state.

In step S107, a maximum of M frame memories from the temporally newestone of the remaining frame memories are set in a readable state. In thismanner, the sequence of frame memory control M is complete, and the flowadvances to step S8 in FIG. 4.

Details of frame memory control N in step S101 will be explained belowwith reference to FIG. 6.

FIG. 6 is a flowchart showing the details of frame memory control Naccording to the second embodiment of the present invention.

In step S110, the number of reference frames is set to N (in this case,2).

In step S111, N frame memories from the temporally newest one of theremaining frame memories are set in a readable state. In step S112, oneof the remaining frame memories is set in a writable state.

In step S113, power supply to the still remaining frame memories isstopped because they are unused frame memories. In this way, thesequence of memory control N is complete, and the flow advances to stepS8.

Referring back to FIG. 4, as in the first embodiment, after processes insteps S8 to S10, in step S102 the decoded image is stored in a writableone of the frame memories. In step S12, the coded data is recorded onthe recording medium 9, and the flow returns to step S2 to process thenext frame.

In the second embodiment as described above, in addition to the effectsexplained in the first embodiment, the power consumption can be reducedby stopping power-supply to unused frame memories. Therefore, especiallyin an environment in which the apparatus is operated by a limited powersupply, the operation time of the apparatus can be prolonged.

In the second embodiment, the maximum number of reference frames is setto 5. However, it is obviously also possible to use any number ofreference frames by using a necessary number of frame memories.

Also, in the second embodiment, the number of reference frames when themotion is large is not limited to the above-mentioned number. Inaddition, the number of frames to be selected may also be variable inaccordance with the magnitude of the motion. For example, the number offrames which can be referred to can be determined inversely proportionalto the magnitude of the motion of the motion image coding apparatus, orsome other standard may also be used.

The second embodiment is explained by taking only forward prediction asan example. However, backward prediction or bidirectional prediction mayalso be performed by using a necessary number of frame memories.

It is, of course, also possible to describe, in software, some or all ofthe various constituent elements of the motion image coding apparatusaccording to the second embodiment, and execute processing by anarithmetic unit such as a CPU.

<Third Embodiment>

FIG. 7 is a block diagram showing the arrangement of a motion imagecoding apparatus according to the third embodiment of the presentinvention.

The same reference numerals as in FIG. 3 of the second embodiment denoteparts having the same functions in FIG. 7, and an explanation thereofwill be omitted.

Referring to FIG. 7, an image sensing mode setting unit 200 sets theimage sensing mode of the motion image coding apparatus. Examples of theimage sensing mode are “automatic mode” as a standard, “sports mode” forsensing an image of an object having a high speed, and “scenery mode”for sensing images of scenes such as mountains, seas, and the like.However, the image sensing modes are not limited to these examples.

A frame memory controller 201 controls frame memories 12 to 17. Inaccordance with the image sensing mode, the frame memory controller 201controls write and read and power supply to the frame memories 12 to 17.An imaging device 202 can control the shutter speed, aperture value, andthe like in accordance with instructions from the image sensing modesetting unit 200.

The motion image coding operation of the motion image coding apparatusshown in FIG. 7 will be described below.

First, a user (not shown) sets the image sensing mode of the imagingdevice 202 by using the image sensing mode setting unit 200 beforestarting image sensing. The set image sensing mode is input to theimaging device 202 and frame memory controller 201. The imaging device202 selects and sets a shutter speed and F-number suited to the setimage sensing mode.

When “automatic mode” is selected, the frame memory controller 201 setsthe number of reference frames to 3, and stops power supply to the framememories 16 and 17 to keep them unused. When “sports mode” is selected,the frame memory controller 201 sets the number of reference frames to1, and stops power supply to the frame memories 14 to 17 to keep themunused. When “scenery mode” is selected, the frame memory controller 201sets the number of reference frames to 5.

After that, intra-frame coding or inter-frame coding of each frame isexecuted by using frame memories to which electric power is supplied.

When intra-frame coding is to be executed, the frame memory controller201 sets, from the frame memories to which power is supplied, an emptyframe memory or a frame memory storing the temporally oldest frame in awritable state in order, and stores the decoded image.

When inter-frame coding is to be executed, the frame memory controller201 sets, from of the frame memories to which power is supplied, anempty frame memory or a frame memory storing the temporally oldest framein a writable state in order, and sets the other frame memories in areadable state.

Note that in inter-frame coding, the number of reference frames is setfor each image sensing mode. Therefore, in accordance with the number ofreference frames, motion vectors are calculated, and motion compensationis executed. The generated coded data is recorded on a recording medium9 via a recorder 8. The decoded image is stored in the frame memoriesset in the writable state.

The process flow of the motion image coding process of the motion imagecoding apparatus according to the third embodiment will be describedbelow with reference to FIG. 8.

FIG. 8 is a flowchart showing the motion image coding process of themotion image coding apparatus according to the third embodiment of thepresent invention.

The same reference numerals as in FIG. 2 of the first embodiment denotesteps having the same functions in FIG. 8, and an explanation thereofwill be omitted.

In step S200, the image sensing mode is set.

Assume that in FIG. 8, “sports mode”, “scenery mode”, and “automaticmode” are the image sensing modes.

In step S201, whether the image sensing mode is “sports mode” isdetermined. If the image sensing mode is “sports mode” (YES in stepS201), the flow advances to step S202 to set the number of referenceframes to N (in this case, 1). In step S203, power supply to unusedframe memories are stopped. After that, the flow advances to step S1.

If the image sensing mode is not “sports mode” in step S201 (NO in stepS201), the flow advances to step S204 to determine whether the imagesensing mode is “scenery mode”. If the image sensing mode is “scenerymode”, the flow advances to step S205 to set the number of referenceframes to P (in this case, 5). After that, the flow advances to step S1.

If the image sensing mode is not “scenery mode” (NO in step S204), theflow advances to step S206 to set the number of reference frames to M(in this case, 3). In step S207, power supply to unused frame memoriesis stopped. The flow then advances to step S1. In this embodiment, P≧M≧Nalways holds.

After that, as in the first embodiment, processes in steps S1 to S3 andS8 to S12 are executed on the basis of one of the set numbers ofreference frames P, M, and N.

In the third embodiment as described above, the number of referenceframes is controlled on the basis of the magnitude, which is expected bythe image sensing mode, of the motion of a motion image to be coded. Asa consequence, optimum processing can be executed in accordance with themotion.

Also, since the number of reference frames is preset on the basis of theimage sensing mode, optimum reference can be performed. In addition, thepower consumption can be reduced by stopping power supply to unusedframe memories. Therefore, especially in an environment in which theapparatus is operated by a limited power supply, the operation time ofthe apparatus can be prolonged.

In the third embodiment, the maximum number of reference frames is setto 5. However, it is obviously also possible to use any number ofreference frames by using a necessary number of frame memories.

Also, the third embodiment is explained by taking only forwardprediction as an example. However, backward prediction or bidirectionalprediction may also be performed by using a necessary number of framememories.

It is, of course, also possible to describe, in software, some or all ofthe various constituent elements of the motion image coding apparatusaccording to the third embodiment, and execute processing by anarithmetic unit such as a CPU.

<Fourth Embodiment>

FIG. 9 is a block diagram showing the arrangement of a motion imagecoding apparatus according to the fourth embodiment of the presentinvention.

Referring to FIG. 9, a central processing unit (CPU) 300 controls theentire apparatus and performs various processes. A memory 301 providesstorage areas for an operating system (OS), software, and arithmeticoperations necessary to control this apparatus. A bus 302 interconnectsvarious constituent elements forming the motion image coding apparatus,and transfers data and control signals.

A terminal 303 is used to activate the apparatus, set variousconditions, and execute playback instructions. A storage device 304stores software. A storage device 305 stores coded streams. The storagedevices 304 and 305 may also be portable storage media which can beseparated from the motion image coding apparatus and moved.

A camera (imaging device) 307 can sense motion images frame by frame. Acamera panhead 306 of the camera 307 is controlled by software, and hasa function of outputting, to the bus 302, image data from the camera307, the state of the camera 307, and the state of the camera panhead306 itself.

A monitor 308 displays images. A communication line 310 is, e.g., a LAN,public line, radio channel, or broadcasting radio wave. A communicationinterface (I/F) 309 transmits and receives streams via the communicationline 310.

The memory 301 stores the OS for controlling the entire apparatus andoperating various software, and also stores the software to be operated.In addition, the memory 301 has an image area for storing image data, acode area for storing generated coded data, and a working area forstoring, e.g., parameters for various arithmetic operations and coding.

A process which, in the arrangement as described above, controls thecamera panhead 306 and the camera 307 from an external terminal via thecommunication line 310, codes a motion image sensed by the camera 307,and transmits the coded motion image to the external terminal via thecommunication line 310, will be described below.

In the fourth embodiment, a coding method used by the motion imagecoding apparatus will be explained by taking the H.264 coding method asan example. However, any coding method can be used as long as an imageis coded by referring to two or more frames. Also, to simplify theexplanation, three frames before a frame to be coded and three framesafter it are referred to. However, the number of reference frames is notlimited.

Before the processing, the terminal 303 instructs the whole apparatus toperform activation, thereby initializing the individual constituentelements. Consequently, software stored in the storage device 304 isdeveloped on the memory 301 via the bus 302, and activated.

The data configuration of the memory 301 will be explained below withreference to FIG. 10.

FIG. 10 is a view showing the data configuration of the memory accordingto the fourth embodiment of the present invention.

As shown in FIG. 10, the memory 301 stores the OS for controlling thewhole apparatus and operating various software, motion image codingsoftware for executing the H.264 coding method, communication softwarefor controlling communication, and camera control software forcontrolling the camera panhead 306. The memory 301 also has image areas1 to 7 for storing image data, a code area for storing coded data, and aworking area.

The way the CPU 300 controls the motion image coding apparatus in thearrangement as described above will be explained below with reference toFIG. 11.

FIG. 11 is a flowchart showing processing executed by the motion imagecoding apparatus according to the fourth embodiment of the presentinvention.

In step S300, the CPU 300 activates the camera control software toinitialize the individual parts, and starts sensing motion images. Instep S301, the CPU 300 activates the communication software to receivecamera control information via the communication line 310, therebymaking preparations for outputting coded data.

In step S302, the CPU 300 determines whether process termination isinput from the terminal 303. If termination is input (YES in step S302),the flow advances to step S307, and the CPU 300 terminates thecommunication software. In step S308, the CPU 300 terminates the cameracontrol software, and then terminates the entire process.

On the other hand, if termination is not input in step S302 (NO in stepS302), the flow advances to step S303, and the CPU 300 determineswhether a camera control request and motion image transmission requestare input from the communication line 310 via the communicationinterface 309. If no such requests are input (NO in step S303), the flowreturns to step S302, and the CPU 300 waits until these requests areinput. If these requests are input (YES in step S303), the flow advancesto step S304.

In step S304, the CPU 300 activates the motion image coding software. Instep S305, the CPU 300 executes coding and transmits the coded data.Details of this process will be described later. When the communicationis complete, the flow advances to step S306. In step S306, the CPU 300terminates the coding software. The flow then returns to step S302, andthe CPU 300 waits for the next instruction.

The details of the process in step S305 will be explained below withreference to FIG. 12.

FIG. 12 is a flowchart showing the details of the process in step S305according to the fourth embodiment of the present invention.

First, in step S351, header information is generated and stored in thecode area on the memory 301. When the coded data is stored in the codearea, the communication software transmits the data to the communicationline 310 via the communication interface 309, and, after thetransmission, clears the corresponding region of the code area.Transmission of the coded data in the code area will not particularly bedescribed in the following explanation.

In step S352, the CPU 300 determines the presence/absence of acommunication termination request. If there is a communicationtermination request (YES in step S352), the CPU 300 terminates theprocess. If there is no communication termination request (NO in stepS352), the flow advances to step S353.

In step S353, if a camera control command for controlling the camera isreceived from the communication line 310 via the communication interface309, the CPU 300 inputs the command to the camera panhead 306. On thebasis of this camera control command, the camera panhead 306 performsoperations such as panning, tilting, and zooming.

In step S354, the CPU 300 inputs frame data taken by the camera 307 viathe camera panhead 306, and stores the data in image area 1 on thememory 301.

In step S355, the CPU 300 acquires data concerning the motion of thecamera 307 from the camera panhead 306. That is, the camera panhead 306operates the camera 307 after receiving the camera control command, anddata concerning the motion of the camera 307 can be acquired bymonitoring the status of a motor or the like. The CPU 300 writes theacquired data in the working area on the memory 301.

In step S356, the CPU 300 compares the obtained data (the magnitude ofthe motion) pertaining to the motion of the camera 307 with apredetermined value.

If the magnitude of the motion is equal to or smaller than thepredetermined value (NO in step S356), the flow advances to step S357,and the CPU 300 sets the number of front and back reference frames to M(in this case, 3). Then, in step S358, the CPU 300 sets m×m as the rangeof search for motion vectors to be used when motion compensation isperformed during inter-frame coding (to be described later). In thiscase, m is, e.g., 32 pixels. After that, the flow advances to step S361.

If the magnitude of the motion is larger than the predetermined value(YES in step S356), the flow advances to step S359, and the CPU 300 setsthe number of front and back reference frames to N (in this case, 1). Inthis embodiment, M>N always holds. In step S360, the CPU 300 sets n×n asthe range of search for motion vectors to be used when motioncompensation is performed during inter-frame coding (to be describedlater). In this case, n is, e.g., 55 pixels. After that, the flowadvances to step S361.

In step S361, the CPU 300 determines the coding mode of frame data to becoded.

If the coding mode is intra-frame coding, the flow advances to stepS362, and the CPU 300 executes intra-frame coding for the input framedata. The CPU 300 then stores the generated coded data in the code areaon the memory 301, and the flow advances to step S364.

If the coding mode is inter-frame coding, the flow advances to stepS363, and the CPU 300 executes inter-frame coding by executing motioncompensation by searching for motion vectors within the set range, whilereferring to an image in an image area having the set number ofreference frames. The CPU 300 then stores the generated coded data inthe code area on the memory 301, and the flow advances to step S364.

In step S364, the CPU 300 stores the decoded image in an empty imagearea or in an image area storing the temporally oldest image data of theimage areas on the memory 301. In step S365, the CPU 300 transmits thecoded data stored in the code area to the communication line 310 via thecommunication interface 309, and the flow returns to step S352 toprocess the next frame.

In the fourth embodiment as described above, if the motion of the camerais large (lively), temporally separated frames are not referred tobecause the reference frequency of these frames extremely decreases. Inthis manner, the arithmetic processing amount of motion compensation canbe greatly reduced. As a consequence, the processing speed can beincreased.

Also, if the motion of the camera is large (lively), accurate motionvector search can be performed by making the motion vector search rangewider than that when the camera motion is small. That is, when thenumber of reference frames is reduced from 5 to 2, the arithmeticoperation amount remains the same even if the motion vector search rangeis extended to (3025(55×55))/1024(32×32))^(1/2). This makes it possibleto efficiently search for motion vectors without increasing the load onthe CPU.

In the fourth embodiment, reference frames are three frames before aframe to be coded and three frames after it. However, it is obviouslyalso possible to use any number of reference frames by securing anecessary amount of image areas for storing reference frames.

In the fourth embodiment, data is transmitted to the communication line310 via the communication interface 309 by activating the communicationsoftware. However, data may also be stored in the storage device 305.

In addition, input images need not be sensed images from the camera. Itis also possible to use motion image data stored in the storage device305, and change the number of reference frames while determining themagnitude of the motion of the frame. Furthermore, the number ofreference frames may also be changed if a scene change or the like isdetected.

It is, of course, also possible to implement embodiments by freelycombining the arrangements of the first to fourth embodiments inaccordance with the applications or purposes.

As many apparently widely different embodiments of the present inventioncan be made without departing from the spirit and scope thereof, it isto be understood that the invention is not limited to the specificembodiments thereof except as defined in the appended claims.

CLAIM OF PRIORITY

This application claims priority from Japanese Patent Application No.2003-425826 filed on Dec. 22, 2003, which is hereby incorporated byreference herein.

1. A motion image coding apparatus which codes a motion image byexecuting motion compensation for frame data to be coded by referring toa plurality of frame data in the motion image, comprising: detectingmeans for detecting a motion of an imaging device; a plurality ofstorage means for storing said plurality of frame data; selecting meansfor selecting, from said plurality of storage means, on the basis ofmotion information detected by said detecting means, storage means forstoring reference frame data to be referred to when the frame data to becoded is coded; estimating means for estimating a motion vector on thebasis of the reference frame data stored in said storage means selectedby said selecting means and the frame data to be coded; coding means forcoding the frame data to be coded by using the motion vector estimatedby said estimating means; and output means for outputting the coded datawhich is coded by said coding means.
 2. The motion image codingapparatus according to claim 1, wherein said detecting means detects themotion of said imaging device on the basis of a motion image sensed bysaid imaging device.
 3. The motion image coding apparatus according toclaim 1, wherein said selecting means comprises control means forcontrolling write/read and power supply to said plurality of storagemeans on the basis of the motion information detected by said detectingmeans.
 4. The motion image coding apparatus according to claim 1,further comprising setting means for setting an image sensing mode ofsaid imaging device, wherein said selecting means comprises controlmeans for controlling write/read and power supply to said plurality ofstorage means on the basis of the image sensing mode set by said settingmeans.
 5. The motion image coding apparatus according to claim 3,wherein said control means stops power supply to storage means notselected by said selecting means.
 6. The motion image coding apparatusaccording to claim 1, further comprising search range control means forcontrolling a motion vector search range of said estimating means on thebasis of the motion image detected by said detecting means.
 7. A motionimage coding apparatus which codes a motion image by executing motioncompensation for frame data to be coded by referring to a plurality offrame data in the motion image, comprising: setting means for setting animage sensing mode of an imaging device; a plurality of storage meansfor storing said plurality of frame data; selecting means for selecting,from said plurality of storage means, on the basis of the image sensingmode set by said setting means, storage means for storing referenceframe data to be referred to when the frame data to be coded is coded;estimating means for estimating a motion vector on the basis of thereference frame data stored in said storage means selected by saidselecting means and the frame data to be coded; coding means for codingthe frame data to be coded by using the motion vector estimated by saidestimating means; and output means for outputting the coded data whichis coded by said coding means.
 8. The motion image coding apparatusaccording to claim 7, wherein said selecting means comprises controlmeans for controlling write/read and power supply to said plurality ofstorage means.
 9. The motion image coding apparatus according to claim8, wherein said control means stops power supply to storage means notselected by said selecting means.
 10. The motion image coding apparatusaccording to claim 7, wherein said detecting means detects a motion ofsaid imaging device on the basis of a motion image sensed by saidimaging device.
 11. A motion image coding apparatus which codes a motionimage by executing motion compensation for frame data to be coded byreferring to a plurality of frame data in the motion image, comprising:input means for inputting control information which controls an imagingdevice; storage means for storing a motion image sensed by said imagingdevice; setting means for setting the number of reference frame data tobe referred to when the frame data to be coded is coded, on the basis ofmotion information of said imaging device, which is acquired on thebasis of the control information input by said input means; acquiringmeans for acquiring reference frame data corresponding to the number ofreference frame data set by said setting means; estimating means forestimating a motion vector on the basis of the reference frame dataacquired by said acquiring means and the frame data to be coded; codingmeans for coding the frame data to be coded by using the motion vectorestimated by said estimating means; and output means for outputting thecoded data which is coded by said coding means.
 12. The motion imagecoding apparatus according to claim 11, further comprising search rangecontrol means for controlling a motion vector search range of saidestimating means on the basis of the motion information.
 13. A controlmethod of a motion image coding apparatus which comprises a plurality ofstorage units for storing a plurality of frame data in a motion image,and codes the motion image by executing motion compensation for framedata to be coded by referring to frame data stored in the plurality ofstorage units, comprising: a detection step of detecting a motion of animaging device; a selection step of selecting, from the plurality ofstorage units, on the basis of motion information detected in thedetection step, a storage unit for storing reference frame data to bereferred to when the frame data to be coded is coded; an estimation stepof estimating a motion vector on the basis of the reference frame datastored in the storage unit selected in the selection step and the framedata to be coded; a coding step of coding the frame data to be coded byusing the motion vector estimated in the estimation step; and an outputstep of outputting the coded data which is coded in the coding step. 14.A control method of a motion image coding apparatus which comprises aplurality of storage units for storing a plurality of frame data in amotion image, and codes a motion image by executing motion compensationfor frame data to be coded by referring to frame data stored in theplurality of storage units, comprising: a setting step of setting animage sensing mode of an imaging device; a selection step of selecting,from the plurality of storage units, on the basis of the image sensingmode set in the setting step, a storage unit for storing reference framedata to be referred to when the frame data to be coded is coded; anestimation step of estimating a motion vector on the basis of thereference frame data stored in the storage unit selected in theselection step and the frame data to be coded; a coding step of codingthe frame data to be coded by using the motion vector estimated in theestimation step; and an output step of outputting the coded data whichis coded in the coding step.
 15. A control method of a motion imagecoding apparatus which comprises a storage unit for storing a motionimage, and codes the motion image by executing motion compensation forframe data to be coded by referring to frame data stored in the storageunit, comprising: an input step of inputting control information whichcontrols an imaging device; a setting step of setting the number ofreference frame data to be referred to when the frame data to be codedis coded, on the basis of motion information of the imaging device,which is acquired on the basis of the control information input in theinput step; an acquisition step of acquiring reference frame datacorresponding to the number of reference frame data set in the settingstep; an estimation step of estimating a motion vector on the basis ofthe reference frame data acquired in the acquisition step and the framedata to be coded; a coding step of coding the frame data to be coded byusing the motion vector estimated in the estimation step; and an outputstep of outputting the coded data which is coded in the coding step. 16.A program for implementing control of a motion image coding apparatuswhich comprises a plurality of storage units for storing a plurality offrame data in a motion image, and codes the motion image by executingmotion compensation for frame data to be coded by referring to framedata stored in the plurality of storage units, comprising program codesof: a detection step of detecting a motion of an imaging device; aselection step of selecting, from the plurality of storage units, on thebasis of motion information detected in the detection step, a storageunit for storing reference frame data to be referred to when the framedata to be coded is coded; an estimation step of estimating a motionvector on the basis of the reference frame data stored in the storageunit selected in the selection step and the frame data to be coded; acoding step of coding the frame data to be coded by using the motionvector estimated in the estimation step; and an output step ofoutputting the coded data which is coded in the coding step.
 17. Aprogram for implementing control of a motion image coding apparatuswhich comprises a plurality of storage units for storing a plurality offrame data in a motion image, and codes a motion image by executingmotion compensation for frame data to be coded by referring to framedata stored in the plurality of storage units, comprising program codesof: a program code of a setting step of setting an image sensing mode ofan imaging device; a program code of a selection step of selecting, fromthe plurality of storage units, on the basis of the image sensing modeset in the setting step, a storage unit for storing reference frame datato be referred to when the frame data to be coded is coded; anestimation step of estimating a motion vector on the basis of thereference frame data stored in the storage unit selected in theselection step and the frame data to be coded; a coding step of codingthe frame data to be coded by using the motion vector estimated in theestimation step; and an output step of outputting the coded data whichis coded in the coding step.
 18. A program for implementing control of amotion image coding apparatus which comprises a storage unit for storinga motion image, and codes the motion image by executing motioncompensation for frame data to be coded by referring to frame datastored in the storage unit, comprising program codes of: an input stepof inputting control information which controls an imaging device; asetting step of setting the number of reference frame data to bereferred to when the frame data to be coded is coded, on the basis ofmotion information of the imaging device, which is acquired on the basisof the control information input in the input step; an acquisition stepof acquiring reference frame data corresponding to the number ofreference frame data set in the setting step; an estimation step ofestimating a motion vector on the basis of the reference frame dataacquired in the acquisition step and the frame data to be coded; acoding step of coding the frame data to be coded by using the motionvector estimated in the estimation step; and an output step ofoutputting the coded data which is coded in the coding step.